System and method for deep learning image super resolution

ABSTRACT

In a method for super resolution imaging, the method includes: receiving, by a processor, a low resolution image; generating, by the processor, an intermediate high resolution image having an improved resolution compared to the low resolution image; generating, by the processor, a final high resolution image based on the intermediate high resolution image and the low resolution image; and transmitting, by the processor, the final high resolution image to a display device for display thereby.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/693,146, filed Nov. 22, 2019, now U.S. Pat. No. 10,970,820, issuedApr. 6, 2021, which is a continuation of U.S. patent application Ser.No. 15/671,036, filed on Aug. 7, 2017, now U.S. Pat. No. 10,489,887,issued Nov. 26, 2019, which claims priority to and the benefit of U.S.Provisional Application No. 62/483,924, entitled “DEEP LEARNING SYSTEMAND METHOD FOR IMAGE SUPER RESOLUTION WITH REAL-TIME PREVIEW,” filed inthe United States Patent and Trademark Office on Apr. 10, 2017, theentire contents of all of which are incorporated herein by reference.

BACKGROUND

Image super-resolution is a process for generating or recovering a highresolution (HR) image from a single low resolution (LR) image. The inputis a blurred or LR image. The output is a high resolution image. Incertain applications, generating a high quality super resolution imagebased on a low resolution input image may be difficult in real time dueto the amount of data processing power and time required to obtain thedesired quality.

The above information disclosed in this Background section is only forenhancement of understanding of the background of the disclosure andtherefore it may contain information that does not form prior art.

SUMMARY

Aspects of some example embodiments of the present disclosure relate toa system and method for deep learning image super resolution.

According to some example embodiments of the present disclosure, in amethod for super resolution imaging, the method includes: receiving, bya processor, a low resolution image; generating, by the processor, anintermediate high resolution image having an improved resolutioncompared to the low resolution image; generating, by the processor, afinal high resolution image based on the intermediate high resolutionimage and the low resolution image; and transmitting, by the processor,the final high resolution image to a display device for display thereby.

According to some example embodiments, the method further includes:receiving, by the processor, a signal from a user device requestingcapture of the intermediate high resolution image; and generating, bythe processor, the final high resolution image after receiving thesignal from the user device.

According to some example embodiments, the method further includes:generating, by the processor, one or more intermediate high resolutionimages concurrently, each utilizing an individual convolution neuralnetwork; and storing, by the processor, the one or more intermediatehigh resolution image in a buffer.

According to some example embodiments, the method further includes:receiving, by the processor, a signal from a user device requestingcapture of the one or more intermediate high resolution image;retrieving, by the processor, the one or more intermediate highresolution images from the buffer; and generating, by the processor, thefinal high resolution image after receiving the signal from the userdevice based on processing the one or more intermediate high resolutionimages from the buffer together with another convolutional neuralnetwork.

According to some example embodiments, the method further includesstoring, by the processor, the low resolution image in a first buffer.

According to some example embodiments, the method further includes:generating, by the processor, the intermediate high resolution imageutilizing a first individual convolution neural network; storing, by theprocessor, the intermediate high resolution image in a second buffer;receiving, by the processor, a signal from a user device requestingcapture of the intermediate high resolution image; retrieving, by theprocessor, the low resolution image from the first buffer; andgenerating, by the processor, another intermediate high resolution imageby a second individual convolution neural network.

According to some example embodiments, the method further includes:applying, by the processor, a first fusion layer with a firstconvolution kernel to an output image of the first individualconvolution neural network to generate a first data set; applying, bythe processor, a second fusion layer with a second convolution kernel toan output image of the second individual convolution neural network togenerate a second data set; and merging, by the processor, the firstdata set and the second data set to generate the final high resolutionimage.

According to some example embodiments, in a system for super resolutionimaging, the system includes: a processor; and a memory coupled to theprocessor, wherein the memory stores instructions that, when executed bythe processor, cause the processor to: receive a low resolution image;generate an intermediate high resolution image having an improvedresolution compared to the low resolution image; generate a final highresolution image based on the intermediate high resolution image and thelow resolution image; and transmit the final high resolution image to adisplay device for display thereby.

According to some example embodiments, the instructions further causethe processor to: receive a signal from a user device requesting captureof the intermediate high resolution image; and generate the final highresolution image after receiving the signal from the user device.

According to some example embodiments, the instructions further causethe processor to: generate one or more the intermediate high resolutionimages concurrently, each utilizing an individual convolution neuralnetwork; and store the one or more intermediate high resolution image ina buffer.

According to some example embodiments, the instructions further causethe processor to: receive a signal from a user device requesting captureof the one or more intermediate high resolution image; retrieve the oneor more intermediate high resolution image from the buffer; and generatethe final high resolution image after receiving the signal from the userdevice based on processing the one or more intermediate high resolutionimages from the buffer together with another convolutional neuralnetwork.

According to some example embodiments, the instructions further causethe processor to store the low resolution image in a first buffer.

According to some example embodiments, the instructions further causethe processor to: generate the intermediate high resolution imageutilizing a first individual convolution neural network; store theintermediate high resolution image in a second buffer; receive a signalfrom a user device requesting capture of the intermediate highresolution image; retrieve the low resolution image from the firstbuffer; and generate another intermediate high resolution image by asecond individual convolution neural network.

According to some example embodiments, the instructions further causethe processor to: apply a first fusion layer with a first convolutionkernel to an output image of the first individual convolution neuralnetwork to generate a first data set; apply a second fusion layer with asecond convolution kernel to an output image of the second individualconvolution neural network to generate a second data set; and merge thefirst data set and the second data set to generate the final highresolution image.

According to some example embodiments, in a system for super resolutionimaging, the system includes: a processor; and a memory coupled to theprocessor, wherein the memory stores instructions that, when executed bythe processor, cause the processor to: receive a low resolution image;generate an intermediate high resolution image having an improvedresolution compared to the low resolution image; generate a final highresolution image based on the intermediate high resolution image and thelow resolution image; receive a signal from a user device requestingcapture of the intermediate high resolution image; generate the finalhigh resolution image after receiving the signal from the user device;and transmit the final high resolution image to a display device fordisplay thereby.

According to some example embodiments, the instructions further causethe processor to: generate the intermediate high resolution imageutilizing an individual convolution neural network; and store theintermediate high resolution image in a buffer.

According to some example embodiments, the instructions further causethe processor to: receive a signal from a user device requesting captureof the intermediate high resolution image; retrieve the intermediatehigh resolution image from the buffer; and generate the final highresolution image after receiving the signal from the user device basedon the intermediate high resolution image from the buffer.

According to some example embodiments, the instructions further causethe processor to store the low resolution image in a first buffer.

According to some example embodiments, the instructions further causethe processor to: generate the intermediate high resolution imageutilizing a first individual convolution neural network; store theintermediate high resolution image in a second buffer; receive a signalfrom a user device requesting capture of the intermediate highresolution image; retrieve the low resolution image from the firstbuffer; and generate another intermediate high resolution image by asecond individual convolution neural network.

According to some example embodiments, the instructions further causethe processor to: apply a first fusion layer with a first convolutionkernel to an output image of the first individual convolution neuralnetwork to generate a first data set; apply a second fusion layer with asecond convolution kernel to an output image of the second individualconvolution neural network to generate a second data set; and merge thefirst data set and the second data set to generate the final highresolution image.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the present disclosure, and many of theattendant features and aspects thereof, will become more readilyapparent as the disclosure becomes better understood by reference to thefollowing detailed description when considered in conjunction with theaccompanying drawings, wherein:

FIG. 1 is a block diagram illustrating aspects of a deep learning imagesuper resolution system, according to some example embodiments of thepresent disclosure;

FIG. 2 is a block diagram illustrating aspects of a progressive fusionsystem according to some example embodiments of the present disclosure;

FIG. 3 is a flow diagram illustrating a process for training aprogressive fusion super resolution imaging system, according to someexample embodiments;

FIG. 4 is an example diagram for pixel wise fusion, according to someexample embodiments;

FIG. 5 is a block diagram of a super resolution imaging system utilizinga context wise fusion architecture, according to some exampleembodiments;

FIG. 6 is a flow diagram illustrating a process for training theparallel context-wise fusion network for dual mode super resolution,according to some example embodiments;

FIG. 7A is a block diagram of a computing device according to anembodiment of the present disclosure;

FIG. 7B is a block diagram of a computing device according to anembodiment of the present disclosure;

FIG. 7C is a block diagram of a computing device according to anembodiment of the present disclosure;

FIG. 7D is a block diagram of a computing device according to anembodiment of the present disclosure; and

FIG. 7E is a block diagram of a network environment including severalcomputing devices according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, example embodiments will be described in more detail withreference to the accompanying drawings, in which like reference numbersrefer to like elements throughout. The present disclosure, however, maybe embodied in various different forms, and should not be construed asbeing limited to only the illustrated embodiments herein. Rather, theseembodiments are provided as examples so that this disclosure will bethorough and complete, and will fully convey some of the aspects andfeatures of the present disclosure to those skilled in the art.Accordingly, processes, elements, and techniques that are not necessaryto those having ordinary skill in the art for a complete understandingof the aspects and features of the present disclosure are not describedwith respect to some of the embodiments of the present disclosure.Unless otherwise noted, like reference numerals denote like elementsthroughout the attached drawings and the written description, and thus,descriptions thereof will not be repeated. In the drawings, the relativesizes of elements, layers, and regions may be exaggerated for clarity.

It will be understood that, although the terms “first,” “second,”“third,” etc., may be used herein to describe various elements,components, regions, layers and/or sections, these elements, components,regions, layers and/or sections should not be limited by these terms.These terms are only used to distinguish one element, component, region,layer or section from another element, component, region, layer orsection. Thus, a first element, component, region, layer or sectiondescribed below could be termed a second element, component, region,layer or section, without departing from the scope of the presentdisclosure.

Spatially relative terms, such as “beneath,” “below,” “lower,” “under,”“above,” “upper,” and the like, may be used herein for ease ofexplanation to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. It will beunderstood that the spatially relative terms are intended to encompassdifferent orientations of the device in use or in operation, in additionto the orientation depicted in the figures. For example, if the devicein the figures is turned over, elements described as “below” or“beneath” or “under” other elements or features would then be oriented“above” the other elements or features. Thus, the example terms “below”and “under” can encompass both an orientation of above and below. Thedevice may be otherwise oriented (e.g., rotated 90 degrees or at otherorientations) and the spatially relative descriptors used herein shouldbe interpreted accordingly. In addition, it will also be understood thatwhen an element or layer is referred to as being “between” two elementsor layers, it can be the only element or layer between the two elementsor layers, or one or more intervening elements or layers may also bepresent.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentdisclosure. As used herein, the singular forms “a,” and “an” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises,” “comprising,” “includes,” and/or “including” when used inthis specification, specify the presence of the stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Expressionssuch as “at least one of,” when preceding a list of elements, modify theentire list of elements and do not modify the individual elements of thelist. Further, the use of “may” when describing embodiments of thepresent disclosure refers to “one or more embodiments of the presentdisclosure.” Also, the term “exemplary” is intended to refer to anexample or illustration.

It will be understood that when an element or layer is referred to asbeing “on,” “connected to,” “connected with,” “coupled to,” or “adjacentto” another element or layer, it can be directly on, connected to,coupled to, or adjacent to the other element or layer, or one or moreintervening elements or layers may be present. When an element or layeris referred to as being “directly on,” “directly connected to,”“directly coupled to,” or “immediately adjacent to” another element orlayer, there are no intervening elements or layers present.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which the present disclosure belongs. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and/orthe present specification, and should not be interpreted in an idealizedor overly formal sense, unless expressly so defined herein.

A super resolution imaging system is an electronic system configured togenerate or recover a high resolution (HR) image based on a single lowresolution (LR) image. Single image super resolution refers to producinga single HR image from one LR image. Multi-image super resolution refersto producing a HR image from multiple images obtained at differenttemporal, angular or spatial dimensions and which need to be aligned insubpixel accuracy.

Super resolution (SR) imaging is becoming increasingly important in avariety of scenarios and applications. For example, super resolutionimaging may be utilized to improve the perceptual quality of web imagesand videos that are present in compressed or down-sampled format. Superresolution imaging may also be utilized in a digital zoom process fordigital cameras, whether standalone or cell phone cameras. Superresolution imaging has additional benefits for a variety of scientificfields and applications that rely on collecting and analyzing images.For example, super resolution imaging may be utilized to improve thequality of microscopy images.

In some applications, real-time performance (e.g., the speed and qualityin which a higher resolution image is generated from a lower resolutionimage) of a super resolution imaging system may be an important factorfor the quality and usefulness of the system. For example, in thecontext of Internet or web browser applications, it may be useful toprovide fast Internet browsing while observing a higher perceptualquality of the images. Fast and high-quality real time super resolutionimaging may also be beneficial for improving the perceived quality ofdigital zoom functionality on the view finder or display of a digitalcamera.

Similarly, in microscopy users may appreciate the benefit of being ableto observe higher resolution images in real-time. For example, in thecontext of light microscopy, a subject that is smaller than a certainthreshold size (e.g., smaller than 250 nanometers (nm) across) mayappear blurred, which may limit or prevent the user from analyzing thesubject in the microscopy image. Thus, super-resolution microscopy mayenable objects to be analyzed at a much smaller scale, with less impactfrom the diffraction limits of the lens. In other circumstances, usersmay benefit from being able to have real time observation of themolecular observations. Thus, in a variety of applications, such asdigital zoom functionality on digital cameras, light microscopy, and webbrowsing, users of a super resolution imaging system may benefit fromhigh quality real time performance, and enhanced offline performance.

Additionally, for the above-mentioned applications, offline performancefor image processing at acquisition time is also of importance. In suchapplications, offline performance can be triggered, for example, byclicking on an image in a browser or saving it, by taking a snapshotfrom the camera using digital zoom, by recording a video or movie withdigital zoom, or by taking a snapshot or recording the microscopeoutput.

Thus, example embodiments of the present disclosure include a system andmethod for super resolution imaging operable in two modes: (1) a previewmode that is fast enough to generate higher resolution images inreal-time (e.g., without any significant perceptible delay) withenhanced quantitative and qualitative perceptual quality compared to alower resolution input image; and (2) an offline mode forpost-processing at acquisition that does not generate an output highresolution image as quickly as the preview mode, but the output imagehas a higher resolution than the image generate in the preview mode.

Some example embodiments of the present disclosure include a system andmethod of providing deep-learning based super-resolution imaging with apreview mode and an offline mode. According to some example embodiments,the computations done in preview mode are leveraged in the offline mode.According to some example embodiments, a system and method includes asuper resolution imaging system that retrieves information fromlow-resolution (LR) images to generate or produce high-resolution (HR)output images. The present system may be achieved by learning very deepconvolution neural networks.

Thus, as described above, some example embodiments of the presentdisclosure include a dual mode SR system and method. According to someexample embodiments, a system and method includes deep learning based SRthat leverages the computations done for real-time SR while in thepreview mode for the offline SR mode. Hence, the present system providesa mechanism for balancing tradeoffs between accuracy and efficiency.

A system and method according to some example embodiments includes apreview network and one or more refine networks. As will be described inmore detail below, some example embodiments may utilize one of two deeplearning architectures described herein: a progressive super resolutionsystem; and a parallel-fused super resolution system.

As described in more detail below, in the progressive super resolutionsystem, the “preview” network and the “refine” networks may be trainedsuccessively. By contrast, in the context-wise parallel fused superresolution system, a context-wise fusion layer for context-wise fusionof the multiple networks in parallel may be utilized. Run-timeprocedures for using the preview networks and the refine network forboth cases are discussed in more detail below.

FIG. 1 is a block diagram illustrating aspects of a deep learning imagesuper resolution system, according to some example embodiments of thepresent disclosure. As illustrated in FIG. 1 , a super resolutionimaging system 100 according to some example embodiments includes animage processing system 102. The image processing system 102 is inelectronic communication with an image source device 104 and an imageoutput device 106. The image source device 104 may be any electronicdevice configured to capture and/or store a digital image, such as adigital microscope, a digital camera, a computer operating an Internetwebsite, and a personal computer system. The image output device 106 maybe any suitable electronic device configured to receive a superresolution image based on a low resolution image. For example, the imageoutput device 106 may be a display device or computer system fordisplaying a high resolution image. As will be described in more detailbelow, the super resolution imaging system 100 is configured to receivea low resolution image from the image source device 104 and generate asuper resolution image based on the low resolution image to provide tothe image output device 106.

According to some example embodiments, systems and methods of thepresent disclosure may utilize one of two deep learning architecturesdescribed herein: a progressive super resolution system; and aparallel-fused super resolution system. FIG. 2 is a block diagramillustrating aspects of a progressive fusion system according to someexample embodiments of the present disclosure.

Referring to FIG. 2 , the super resolution imaging system 100 mayinclude a progressive fusion SR imaging system 200, configured toprovide progressive super resolution, via progressive fusion, to allowfor super resolution imaging to generate an intermediate high resolutionimage as a preview image in a preview mode, and to generate a highresolution image (e.g., having a higher resolution than the intermediatehigh resolution image) in a non-preview or offline mode.

As illustrated in FIG. 2 , the progressive fusion SR imaging system 200includes a first individual super resolution network S₁. The firstindividual network S₁ is configured to receive an LR input image 202(from the image source device 104) and generate an intermediate qualityhigh resolution image (e.g., as a preview image) for providing to adisplay device and/or the image destination device 106. The firstindividual super resolution network S₁ may generate the intermediatequality high resolution image using any suitable super resolutionimaging algorithm or process known in the art.

According to some example embodiments, the first individual superresolution network S₁ includes any suitable convolution neural networkarchitecture. According to some example embodiments, the firstindividual super resolution network S₁ may have relatively lowcomplexity (e.g., having relatively low computational costs ingenerating the output data). The output of the neural network is a feedforward process based on multiplication-and-accumulations of input(e.g., the LR input image) and the weights (e.g., network parameters).If the network is smaller (e.g., because there are relatively fewerlayers and filters), the computational costs will be smaller.Accordingly, the network S₁ can generate the intermediate highresolution image 204 based on the input image in real time (e.g., withlittle or no perceived delay in generating and displaying the image).The baseline of “real time” according to embodiments of the presentdisclosure is 15 frames per second. Thus, according to some exampleembodiments of the present disclosure, the processing time forgenerating an intermediate output image by the first individual superresolution network S₁ is less than 1/15 of a second. Under suchconditions, a human eye will not perceive any delay.

According to some example embodiments, the first individual superresolution network S₁ receives the LR input image 202, or alternatively,a bicubic upsampled version of the LR input image 202, where theupsampling ratio is according to a target upsampling ratio. The outputy₁ of S₁ given the input x₁ can be represented by equation (1) below:y ₁ =S ₁(x ₁).  (1)

The first individual super resolution network S₁ is trained to providean output y₁ that has enough super resolution perceptual quality forenhanced user experience or user perception of the quality of the image.For example, for a camera application, the intermediate high resolutionimage 204 may be a zoomed-in version of the LR input image 202 (e.g., ata scale that is a multiple (e.g., 3 times the size) of the size of theLR input image 202), with the same or higher resolution as the inputimage 202, around an area of interest in the raw image as from theoptical lens. Such increased scaling and image resolution may enable,for example, a user to scan for, and increase the clarity of, objects ofinterest in the zoomed image, read small text, and the like. Thus, theintermediate high resolution image 204 may be provided to and displayedon a device operated by a user that is in electronic communication withor incorporated as part of the super resolution imaging system 100.

According to some example embodiments, once the user is satisfied with acurrent frame and wants to capture the image, the user may transmit asignal (e.g., by selecting a button or prompt in a user interface forinteracting with the progressive fusion SR imaging system 200) to theprogressive fusion SR imaging system 200 to generate the high resolutionimage 206.

The intermediate high resolution image 204 (e.g., the output y₁) mayalso be stored in a temporal frame buffer 208 upon being generated bythe first individual super resolution network S₁ Then, in response toreceiving the signal from the user (or a device operated by the user)indicating the user desires to capture the image, the intermediate highresolution image 204 (e.g., the output y₁) is retrieved from thetemporal frame buffer 208, and provided to a second individual superresolution network S₂ to generate the high resolution image 206.

In particular, the super resolution image generated as output from thepreview network S₁ is used as an intermediate high resolution imagerpreview and is buffered in a temporal frame buffer 208 for further superresolution imaging to generate a final high resolution image based onthe original low resolution input image and the intermediate highresolution image. If the frame is captured, the super resolution outputof the first individual super resolution network S₁ is provided as inputto the second individual super resolution network S₂, to provide anenhanced super resolution quality at the same desired scale. The inputand output resolutions of the second individual super resolution networkS₂ are the same, but the output of the network S₂ has a betterperceptual quality.

For example, according to some example embodiments, the output of thesecond individual super resolution network S₂ e.g., has a higher PeakSignal-to-Noise Ratio (PSNR) or a higher Structural Similarity Measure(SSIM)).

PSNR is the ratio between the maximum possible power of an image pixeland the power of corrupting noise that affects the fidelity. PSNR may becalculated according to equation (2), below:

$\begin{matrix}{{PSNR} = {20\log_{10}\frac{255}{\sqrt{MSE}}}} & (2)\end{matrix}$

In equation (2), the MSE is calculated between the ground truth and areconstructed image (SR output). Larger PSNR corresponds to betterquality. The maximum value of PSNR is infinite.

SSIM is a perception-based model that considers image degradation asperceived change in structural information, while also incorporating theluminance masking and contrast masking. It shows better consistency tohuman vision compared to PSNR. SSIM may be calculated according toequation (3), below:

$\begin{matrix}{{SSIM} = {20\log_{10}\frac{( {{2\mu_{x}\mu_{y}} + c_{1}} )( {{2\sigma_{xy}} + c_{2}} )}{( {\mu_{x}^{2} + \mu_{y}^{2} + c_{1}} )( {\sigma_{x}^{2} + \sigma_{y}^{2} + c_{2}} )}}} & (3)\end{matrix}$

where x is the reconstructed image, y is the reference image (groundtruth), μ is the mean, σ is the variance, and σ_(xy) is the covariancebetween x and y. c₁=6.5025, c₂=58.5225. SSIM lays between [0,1]. If thex is a perfect copy of y, the SSIM will be 1.

The high resolution image 206 may be generated by the second individualsuper resolution network S₂ based on the intermediate high resolutionimage 204 and the LR input image 202 using any suitable super resolutionimaging algorithm or process known in the art.

The output of second stage can be represented by the following equation(4), below:y ₂ =S ₂(S ₁(x ₁))  (4)

Because this is performed offline, more processing power and latency canbe tolerated before the output is stored in memory.

According to some example embodiments, multiple progressive superresolution stages (e.g., additional individual super resolution networksS_(a), where “a” is a natural number greater than 2) can be cascadedwith the first and second individual super resolution networks S₁ andS₂, to get progressively better super resolution quality, with theoutput of each stage provided as input to the next stage. For example, asecond stage may be executed on a mobile device, while a third stage maybe executed by offline software operating on an external computingmachine or a cloud server having higher data processing power andthroughput. Each output from each stage is a meaningful output that canbe considered a super resolution output image. If the output from aparticular stage is satisfactory or desired, a subsequent stage may notbe invoked. A desired image may be based on a number of factors, such asperceptual quality, output generation speed, computation power, or astorage memory requirement, according to the design and function of thesuper resolution imaging system 100 and the needs and desires ofindividual users.

A process for training the progressive fusion SR imaging system 200operating as part of the super resolution imaging system 100, accordingto some example embodiments, is illustrated in FIG. 3 . First, at 300,the super resolution imaging system 100 trains a first network (e.g.,first individual super resolution network S₁) using pairs of patchesfrom input low resolution images downscaled to a target scale and outputground truth images. Second, at 302, the super resolution imaging system100 generates a new dataset, having pairs of patches from theintermediate high resolution output from the first network and theground truth. Third, at 304, the super resolution imaging system 100then modifies a subsequent stage network to perform appropriate resizingat output of each convolutional layer (as stretching or padding) toenforce same input and output sizes. Fourth, at 306, the superresolution imaging system 100 then trains a subsequent stage networkusing the new dataset. Then, at 308, the super resolution imaging system100 determines whether or not an output image having a desiredresolution and perceived quality (e.g., a predefined threshold quality)has generated. If not, the super resolution imaging system 100 repeats304 and 306 until a desired resolution and perceived quality isgenerated, at which the image is provided as an HR output image.

For multiple stages with common network architecture, the networks canbe initialized by that of a previous stage when training the next stagenetwork, which may speed up the convergence of training a subsequentstage network.

According to some example embodiments, a super resolution imaging system100 having a progressive 3-layer and 5-layer network structure mayprovide a relatively better peak signal-to-noise ratio (PSNR) andstructure similarity measure (SSIM) as compared to non-progressive3-layer and 5-layer network structure. The super resolution imagingsystem 100 described above may further be applied to context-wise fusednetwork.

According to some example embodiments, the super resolution imagingsystem 100 provides parallel fusion to achieve dual mode superresolution with deep learning. If the offline step includes multiplestages in the progressive super resolution system, a latency may beintroduced. In some instances, such latency may be desired, because thenetworks of the different stages may not be collocated.

However, if the networks are collocated, the super resolution imagingsystem 100 may invoke multiple stages of the capture mode in parallel.In such embodiments, super resolution imaging system 100 may fuse themultiple stages of the capture step (e.g., the step of generating thefinal high resolution output image using the second individual superresolution network S₂ after the user chooses to capture the highresolution image based on the preview image) in parallel with the outputof the preview stage. Assuming sufficient parallel processing power, thelatency with parallel processing is limited to that of the slowest fusednetwork in the capture stage.

According to some embodiments, the super resolution imaging system 100may provides pixel wise fusion for parallel fusion at the capture step.For example, all of the super resolution networks are trained for thesame ground truth and have the same image size at their output. Forexample, if S_(i)(x_(i)) is the output of the ith network, then withpixel-wise fusion, the output at the (u, v)th pixel of the parallelfused network is a weighted sum as can be calculated according toequation (5), below:y _(u,v)=Σ_(j) w _(j) S _(j,(u,v)) +b _(j)  (5)

FIG. 4 is an example diagram for pixel wise fusion, according to someexample embodiments, in which a pixel-wise weighted sum is applied tothe output image of a plurality of individual super resolution networks,and the sum of each of which is provided as the output image. Forapplications such as super resolution, looking at a pixel independentlyof its neighbors may not be very useful. Thus, according to oneembodiment, the present system and method incorporates contextualinformation about the pixel as predicted from each network beforeproducing the fused output. This may be achieved by fusion using aconvolution layer.

A related convolutional layer applies a three-dimensional (3D) kernel,where the size of the 3^(rd) dimension is the number of input channelsin each layer. In the pixel-wise fused network, the output for a givenpixel is the weighted sum across the convoluted output, i.e., theweighted sum of the surroundings of this pixels, which gives thecontextual information, from all fused networks as determined by thesize of the kernel. Moreover, the additional convolutional layers can beinserted to further get a weighted sum of the features obtained afterfusions.

FIG. 5 is a block diagram of a super resolution imaging system 100utilizing a context wise fusion architecture 350, according to someexample embodiments. Similar to progressive super resolution imaging asdescribed above with respect to FIG. 2 , a buffer 352 is used to bufferthe intermediate high resolution output of an individual preview networkS₁. However, another buffer 354 may be utilized to buffer the originallow resolution frame as well, to feed to the other super resolutionimaging networks in parallel at the capture or offline mode.

As illustrated in FIG. 5 , a first individual super resolution networkS₁ operating as a preview step network may receive an LR input image202, as described above with respect to FIG. 2 . Additionally, the LRinput image 202 is stored in a low resolution frame buffer 354. Theoutput image 356 of the first individual super resolution network S₁ isthen stored as an intermediate high resolution preview image in the highresolution frame buffer 352. Upon receiving a capture signal from a userdevice indicating the user wishes to generate a final high resolutionimage of the preview image, one or more capture step networks (e.g.,individual super resolution networks S₂ and S₃) retrieve the LR inputimage 202 from the low resolution frame buffer 354 to generaterespective intermediate high resolution images 358 and 360. Theintermediate high resolution images 356, 358, and 360 may be generatedusing any suitable super resolution imaging algorithm or process knownin the art. Then corresponding fusion layers with convolution kernelsare applied to the intermediate high resolution images 356, 358, and360, and the output of each is merged into a high resolution image 206to be provided to the output device 106. The merging is based on thecontext-wise fusion, which first convolves the high resolution images356, 358, and 360 by individual fusion layer (W1, b1)(W2, b2)(W3, b3)and then sums them together to obtain the refined HR image 206. Similarto equation (5), if S_(i)(x_(i)) is the output of the ith network, thenwith context fusion, the output at the (u, v)th pixel is a weighted sumas can be calculated according to equation (5). The weighted sum isimplemented implicitly with the context convolutional layer, having ak×k×M convolutional kernel, where M is the number of networks to befused (M=3). In this example, k×k is the context fusion receptive field,where (y,x) pixel of output channel Y_(o) is given by equation (6),below:Y _(o)(y,x)=Σ_(m=1) ^(M)Σ_(r=0) ^(k-1)Σ_(s=0) ^(k-1)W(m,r,s)I(m,y+r,x+s)  (6)

Because the receptive field is much larger compared to pixel-wisefusion, the perceptive quality of the output HR is better.

A process for training the parallel context-wise fusion network for dualmode super resolution, according to some example embodiments, isillustrated in FIG. 6 . First, at 600, the super resolution imagingsystem 100 trains all different networks using pairs of patches frominput low resolution images downscaled to target scale and output groundtruth images. Second, at 602, the super resolution imaging system 100constructs a new network as shown with one or more context fusion layersutilizing 3 dimensional convolutional kernels, appended to the output ofthe 3 networks, initialized by Gaussian distribution.

Then, at 604, according to some embodiments, the super resolutionimaging system 100 may retrain with the same input output pairs, whilefreezing weights of all individual super resolution networks andchanging the weights of the context wise fusion layers. Alternatively,according to some embodiments, the super resolution imaging system 100retrains with the same input output pairs, while allowing fine tuning ofthe parameters of all layers except for those of the preview network.

It can be observed that the training procedure is faster thanprogressive super resolution imaging. In the second scenario, in whichthe super resolution imaging system 100 retrains with the same inputoutput pairs, the weights of the preview network may be frozen toleverage the computations done during the preview mode. The output fromthe preview stage is fed directly into the context wise fusion stage andthe preview network need not be rerun again.

The application of progressive super resolution imaging or parallelcontext-wise fusion depends on the objectives of the user and the designand function of the super resolution imaging system 100. If all networksare collocated, the best performance may be achieved with parallelcontext wise fusion. Otherwise, for delayed application of the nextstages, the progressive super resolution fusion architecture may achievesuperior results.

According to some example embodiments, a super resolution imaging system100 provides cascade fusion including a preview network and at least onerefinement network, a method for training the networks, and a method foroperation of progressive super resolution imaging. According to anotherembodiment, the super resolution imaging system 100 provides parallelfusion including a preview network, at least one refinement network, andcontext-wise fusion layers, a method for training the networks and theirfusion layers, and a method for operation in preview mode and refinementmodes.

FIG. 7A and FIG. 7B depict block diagrams of a computing device 1500 asmay be employed in example embodiments of the present disclosure. Forexample, the computing device 1500 may be utilized for the variouscomponents of the super resolution imaging system 100.

Each computing device 1500 includes a central processing unit 1521 and amain memory unit 1522. As shown in FIG. 7A, the computing device 1500may also include a storage device 1528, a removable media interface1516, a network interface 1518, an input/output (I/O) controller 1523,one or more display devices 1530 c, a keyboard 1530 a and a pointingdevice 1530 b, such as a mouse. The storage device 1528 may include,without limitation, storage for an operating system and software. Asshown in FIG. 7B, each computing device 1500 may also include additionaloptional elements, such as a memory port 1503, a bridge 1570, one ormore additional input/output devices 1530 d, 1530 e and a cache memory1540 in communication with the central processing unit 1521. Theinput/output devices 1530 a, 1530 b, 1530 d, and 1530 e may collectivelybe referred to herein using reference numeral 1530.

The central processing unit 1521 is any logic circuitry that responds toand processes instructions fetched from the main memory unit 1522. Itmay be implemented, for example, in an integrated circuit, in the formof a microprocessor, microcontroller, or graphics processing unit (GPU),or in a field-programmable gate array (FPGA) or application-specificintegrated circuit (ASIC). The main memory unit 1522 may be one or morememory chips capable of storing data and allowing any storage locationto be directly accessed by the central processing unit 1521. As shown inFIG. 7A, the central processing unit 1521 communicates with the mainmemory unit 1522 via a system bus 1550. As shown in FIG. 7B, the centralprocessing unit 1521 may also communicate directly with the main memoryunit 1522 via a memory port 1503.

FIG. 7B depicts an embodiment in which the central processing unit 1521communicates directly with cache memory 1540 via a secondary bus,sometimes referred to as a backside bus. In other embodiments, thecentral processing unit 1521 communicates with the cache memory 1540using the system bus 1550. The cache memory 1540 typically has a fasterresponse time than main memory unit 1522. As shown in FIG. 7A, thecentral processing unit 1521 communicates with various I/O devices 1530via the local system bus 1550. Various buses may be used as the localsystem bus 1550, including a Video Electronics Standards Association(VESA) Local bus (VLB), an Industry Standard Architecture (ISA) bus, anExtended Industry Standard Architecture (EISA) bus, a MicroChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI Extended (PCI-X) bus, a PCI-Express bus, or a NuBus. For embodimentsin which an I/O device is a display device 1530 c, the centralprocessing unit 1521 may communicate with the display device 1530 cthrough an Advanced Graphics Port (AGP). FIG. 7B depicts an embodimentof a computer 1500 in which the central processing unit 1521communicates directly with I/O device 1530 e. FIG. 7B also depicts anembodiment in which local busses and direct communication are mixed: thecentral processing unit 1521 communicates with I/O device 1530 d using alocal system bus 1550 while communicating with I/O device 1530 edirectly.

A wide variety of I/O devices 1530 may be present in the computingdevice 1500. Input devices include one or more keyboards 1530 a, mice,trackpads, trackballs, microphones, and drawing tablets. Output devicesinclude video display devices 1530 c, speakers, and printers. An I/Ocontroller 1523, as shown in FIG. 7A, may control the I/O devices. TheI/O controller may control one or more I/O devices such as a keyboard1530 a and a pointing device 1530 b, e.g., a mouse or optical pen.

Referring again to FIG. 7A, the computing device 1500 may support one ormore removable media interfaces 1516, such as a floppy disk drive, aCD-ROM drive, a DVD-ROM drive, tape drives of various formats, a USBport, a Secure Digital or COMPACT FLASH™ memory card port, or any otherdevice suitable for reading data from read-only media, or for readingdata from, or writing data to, read-write media. An I/O device 1530 maybe a bridge between the system bus 1550 and a removable media interface1516.

The removable media interface 1516 may for example be used forinstalling software and programs. The computing device 1500 may furthercomprise a storage device 1528, such as one or more hard disk drives orhard disk drive arrays, for storing an operating system and otherrelated software, and for storing application software programs.Optionally, a removable media interface 1516 may also be used as thestorage device. For example, the operating system and the software maybe run from a bootable medium, for example, a bootable CD.

In some embodiments, the computing device 1500 may comprise or beconnected to multiple display devices 1530 c, which each may be of thesame or different type and/or form. As such, any of the I/O devices 1530and/or the I/O controller 1523 may comprise any type and/or form ofsuitable hardware, software, or combination of hardware and software tosupport, enable or provide for the connection to, and use of, multipledisplay devices 1530 c by the computing device 1500. For example, thecomputing device 1500 may include any type and/or form of video adapter,video card, driver, and/or library to interface, communicate, connect orotherwise use the display devices 1530 c. In one embodiment, a videoadapter may comprise multiple connectors to interface to multipledisplay devices 1530 c. In other embodiments, the computing device 1500may include multiple video adapters, with each video adapter connectedto one or more of the display devices 1530 c. In some embodiments, anyportion of the operating system of the computing device 1500 may beconfigured for using multiple display devices 1530 c. In otherembodiments, one or more of the display devices 1530 c may be providedby one or more other computing devices, connected, for example, to thecomputing device 1500 via a network. These embodiments may include anytype of software designed and constructed to use the display device ofanother computing device as a second display device 1530 c for thecomputing device 1500. One of ordinary skill in the art will recognizeand appreciate the various ways and embodiments that a computing device1500 may be configured to have multiple display devices 1530 c.

A computing device 1500 of the sort depicted in FIG. 7A and FIG. 7B mayoperate under the control of an operating system, which controlsscheduling of tasks and access to system resources. The computing device1500 may be running any operating system, any embedded operating system,any real-time operating system, any open source operating system, anyproprietary operating system, any operating systems for mobile computingdevices, or any other operating system capable of running on thecomputing device and performing the operations described herein.

The computing device 1500 may be any workstation, desktop computer,laptop or notebook computer, server machine, handheld computer, mobiletelephone or other portable telecommunication device, media playingdevice, gaming system, mobile computing device, or any other type and/orform of computing, telecommunications or media device that is capable ofcommunication and that has sufficient processor power and memorycapacity to perform the operations described herein. In someembodiments, the computing device 1500 may have different processors,operating systems, and input devices consistent with the device.

In other embodiments the computing device 1500 is a mobile device, suchas a Java-enabled cellular telephone or personal digital assistant(PDA), a smart phone, a digital audio player, or a portable mediaplayer. In some embodiments, the computing device 1500 comprises acombination of devices, such as a mobile phone combined with a digitalaudio player or portable media player.

As shown in FIG. 7C, the central processing unit 1521 may includemultiple processors P1, P2, P3, P4, and may provide functionality forsimultaneous execution of instructions or for simultaneous execution ofone instruction on more than one piece of data. In some embodiments, thecomputing device 1500 may comprise a parallel processor with one or morecores. In one of these embodiments, the computing device 1500 is ashared memory parallel device, with multiple processors and/or multipleprocessor cores, accessing all available memory as a single globaladdress space. In another of these embodiments, the computing device1500 is a distributed memory parallel device with multiple processorseach accessing local memory only. In still another of these embodiments,the computing device 1500 has both some memory which is shared and somememory which may only be accessed by particular processors or subsets ofprocessors. In still even another of these embodiments, the centralprocessing unit 1521 comprises a multicore microprocessor, whichcombines two or more independent processors into a single package, e.g.,into a single integrated circuit (IC). In one example embodiment,depicted in FIG. 7D, the computing device 1500 includes at least onecentral processing unit 1521 and at least one graphics processing unit1521′.

In some embodiments, a central processing unit 1521 provides singleinstruction, multiple data (SIMD) functionality, e.g., execution of asingle instruction simultaneously on multiple pieces of data. In otherembodiments, several processors in the central processing unit 1521 mayprovide functionality for execution of multiple instructionssimultaneously on multiple pieces of data (MIMD). In still otherembodiments, the central processing unit 1521 may use any combination ofSIMD and MIMD cores in a single device.

A computing device may be one of a plurality of machines connected by anetwork, or it may comprise a plurality of machines so connected. FIG.7E shows an example network environment. The network environmentcomprises one or more local machines 1502 a, 1502 b (also generallyreferred to as local machine(s) 1502, client(s) 1502, client node(s)1502, client machine(s) 1502, client computer(s) 1502, client device(s)1502, endpoint(s) 1502, or endpoint node(s) 1502) in communication withone or more remote machines 1506 a, 1506 b, 1506 c (also generallyreferred to as server machine(s) 1506 or remote machine(s) 1506) via oneor more networks 1504. In some embodiments, a local machine 1502 has thecapacity to function as both a client node seeking access to resourcesprovided by a server machine and as a server machine providing access tohosted resources for other clients 1502 a, 1502 b. Although only twoclients 1502 and three server machines 1506 are illustrated in FIG. 7E,there may, in general, be an arbitrary number of each. The network 1504may be a local-area network (LAN), e.g., a private network such as acompany Intranet, a metropolitan area network (MAN), or a wide areanetwork (WAN), such as the Internet, or another public network, or acombination thereof.

The computing device 1500 may include a network interface 1518 tointerface to the network 1504 through a variety of connectionsincluding, but not limited to, standard telephone lines, local-areanetwork (LAN), or wide area network (WAN) links, broadband connections,wireless connections, or a combination of any or all of the above.Connections may be established using a variety of communicationprotocols. In one embodiment, the computing device 1500 communicateswith other computing devices 1500 via any type and/or form of gateway ortunneling protocol such as Secure Socket Layer (SSL) or Transport LayerSecurity (TLS). The network interface 1518 may comprise a built-innetwork adapter, such as a network interface card, suitable forinterfacing the computing device 1500 to any type of network capable ofcommunication and performing the operations described herein. An I/Odevice 1530 may be a bridge between the system bus 1550 and an externalcommunication bus.

According to one embodiment, the network environment of FIG. 7E may be avirtual network environment where the various components of the networkare virtualized. For example, the various machines 1502 may be virtualmachines implemented as a software-based computer running on a physicalmachine. The virtual machines may share the same operating system. Inother embodiments, different operating systems may be run on eachvirtual machine instance. According to one embodiment, a “hypervisor”type of virtualization is implemented where multiple virtual machinesrun on the same host physical machine, each acting as if it has its owndedicated box. Of course, the virtual machines may also run on differenthost physical machines.

Although this disclosure has been described in certain specificembodiments, those skilled in the art will have no difficulty devisingvariations to the described embodiment, which in no way depart from thescope of the present disclosure. Furthermore, to those skilled in thevarious arts, the disclosure itself herein will suggest solutions toother tasks and adaptations for other applications. It is theapplicant's intention to cover by claims all such uses of the disclosureand those changes and modifications which could be made to theembodiments of the disclosure herein chosen for the purpose ofdisclosure without departing from the scope of the disclosure. Thus, thepresent embodiments of the disclosure should be considered in allrespects as illustrative and not restrictive, the scope of thedisclosure to be indicated by the appended claims and their equivalentsrather than the foregoing description.

What is claimed is:
 1. A method for super resolution imaging, the methodcomprising: receiving, by a processor, a low resolution image;generating, by the processor, a first intermediate high resolution imagehaving an improved resolution compared to the low resolution image,wherein the first intermediate high resolution image is generated by afirst super resolution network using the low resolution image;generating, by the processor, a second intermediate high resolutionimage having an improved resolution compared to the low resolutionimage, wherein the second intermediate high resolution image isgenerated by a second super resolution network using the low resolutionimage; merging, by the processor, the first intermediate high resolutionimage and the second intermediate high resolution image; and generating,by the processor, a final high resolution image based on the merging,wherein the generating of the final high resolution image includesperforming a weighted sum across a convoluted output based on the firstsuper resolution network and the second super resolution network.
 2. Themethod of claim 1, wherein the first intermediate high resolution imageis generated concurrently with the second intermediate high resolutionimage.
 3. The method of claim 1, wherein the first intermediate highresolution image and the second intermediate high resolution image aredifferent from an intermediate high resolution preview image.
 4. Themethod of claim 1, further comprising: receiving, by the processor, acapture signal from a user device initiating the generating of the finalhigh resolution image.
 5. The method of claim 1, wherein the generating,by the processor, of the final high resolution image based on themerging comprises: applying, by the processor, a first fusion layercomprising a first convolution kernel to the first intermediate highresolution image; applying, by the processor, a second fusion layercomprising a second convolution kernel to the second intermediate highresolution image; and summing, by the processor, outputs of each of thefirst and second fusion layers to generate the final high resolutionimage.
 6. The method of claim 5, wherein each of the first and secondfusion layers correspond to a context convolutional layer comprising ak×k×M convolutional kernel, wherein M corresponds to a number ofnetworks to be merged, and k×k corresponds to a context fusion receptivefield.
 7. The method of claim 6, wherein the weighted sum is implementedimplicitly by the context convolutional layer.
 8. The method of claim 1,wherein the merging is based on a context-wise fusion.
 9. The method ofclaim 1, further comprising: training, by the processor, each of thefirst and second super resolution networks using pairs of patches frominput low resolution images downscaled to a target scale and outputground images.
 10. The method of claim 9, wherein the training furthercomprises: constructing, by the processor, one or more context fusionlayers comprising a 3-dimensional convolutional kernel; assigning, bythe processor, the one or more context fusion layers to outputs of eachof the first and second super resolution networks; freezing, by theprocessor, weights of each of the first and second super resolutionnetworks; modifying, by the processor, weights of each of the one ormore context fusion layers; and retraining, by the processor, each ofthe first and second super resolution networks using the same pairs ofpatches.
 11. A system for super resolution imaging, the systemcomprising: one or more processors; and memory coupled to the one ormore processors and having instructions stored thereon that causes theone or more processors to: receive a low resolution image; generate afirst intermediate high resolution image having an improved resolutioncompared to the low resolution image, wherein the first intermediatehigh resolution image is generated by a first super resolution networkusing the low resolution image; generate a second intermediate highresolution image having an improved resolution compared to the lowresolution image, wherein the second intermediate high resolution imageis generated by a second super resolution network using the lowresolution image; merge the first intermediate high resolution image andthe second intermediate high resolution image; and generate a final highresolution image based on the merging, wherein the generating of thefinal high resolution image includes performing a weighted sum across aconvoluted output based on the first super resolution network and thesecond super resolution network.
 12. The system of claim 11, wherein thefirst intermediate high resolution image is generated concurrently withthe second intermediate high resolution image.
 13. The system of claim11, wherein the first intermediate high resolution image and the secondintermediate high resolution image are different from an intermediatehigh resolution preview image.
 14. The system of claim 11, wherein theinstructions further cause the one or more processors to: receive acapture signal from a user device initiating the generating of the finalhigh resolution image.
 15. The system of claim 11, wherein to generatethe final high resolution image based on the merging, the instructionsfurther cause the one or more processors to: apply a first fusion layercomprising a first convolution kernel to the first intermediate highresolution image; apply a second fusion layer comprising a secondconvolution kernel to the second intermediate high resolution image; andsum outputs of each of the first and second fusion layers to generatethe final high resolution image.
 16. The system of claim 15, whereineach of the first and second fusion layers correspond to a contextconvolutional layer comprising a k×k×M convolutional kernel, wherein Mcorresponds to a number of networks to be merged, and k×k corresponds toa context fusion receptive field.
 17. The system of claim 16, whereinthe weighted sum is implemented implicitly by the context convolutionallayer.
 18. The system of claim 17, wherein the merging is based on acontext-wise fusion.
 19. The system of claim 11, wherein theinstructions further cause the one or more processors to: train each ofthe first and second super resolution networks using pairs of patchesfrom input low resolution images downscaled to a target scale and outputground images.
 20. The system of claim 19, wherein to train, theinstructions further cause the one or more processors to: construct oneor more context fusion layers comprising a 3-dimensional convolutionalkernel; assign the one or more context fusion layers to outputs of eachof the first and second super resolution networks; freeze weights ofeach of the first and second super resolution networks; modify weightsof each of the one or more context fusion layers; and retrain each ofthe first and second super resolution networks using the same pairs ofpatches.